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[57] ABSTRACT 

The present invention is generally concerned with electronic 
vision devices and methods, and is specifically concerned 
with image augmentation in combination with navigation, 
position, and attitude devices. In the simplest form, devices 
of the invention can be envisioned to include six major 
components: A 1) camera to collect optical information 
about a real scene and present that information as an 
electronic signal to; a 2) computer processor; a 3) device to 
measure the position of the camera; and a 4) device to 
measure the attitude of the camera (direction of the optic 
axis), thus uniquely identifying the scene being viewed, and 
thus identifying a location in; a 5) data base where infor- 
mation associated with various scenes is stored, the com- 
puter processor combines the data from the camera and the 
data base and perfects a single image to be presented at; a 6) 
display whose image is continuously aligned to the real 
scene as it is viewed by the user. 

The present invention is a vision system including devices 
and methods of augmented reality wherein an image of some 
real scene is altered by a computer processor to include 
information from a data base having stored information of 
that scene in a storage location that is identified by the real 
time position and attitude of the vision system. It is a 
primary function of the vision system of the invention, and 
a contrast to the prior art, to present augmented real images 
and data that is continuously aligned with the real scene as 
that scene is naturally viewed by the user of the vision 
system. An augmented image is one that represents a real 
scene but has deletions, additions and supplements. / 

5 Claims, 6 Drawing Sheets 
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ELECTRO-OPTIC VISION SYSTEM WHICH 
EXPLOITS POSITION AND ATTITUDE 

BACKGROUND OF THE INVENTION 

The present invention is generally concerned with elec- 
tronic vision devices and methods, and is specifically con- 
cerned with image augmentation in combination with 
navigation, position, and attitude devices. 

One may have to look quite far into the annals of history 
to find the first uses of maps. Maps generally provide 
information to alert a user to things that are not readily 
apparent from simple viewing of a real scene from the users 
location. For example, a user of a city road map may not be 
able to see a tunnel on Elm street if the user is currendy 
seven miles away on First street and looking in the direction 
of the Elm street tunnel. However, from the First street 
location, the user could determine from a road map that there 
is a tunnel on Elm street. He could learn that the tunnel is 
three miles long, starts on Eighth street and ends on Eleventh 
street. There may even be an indication of the size of the 
tunnel such that it could accommodate four traffic lanes and 
a bicycle lane. 

Unfortunately, it is not always possible to translate the 
information from a map to the real scene that the information 
represents as the scene is actually viewed. It is common for 
users of maps to attempt to align the map to reality to get a 
better "feel" of where things are in relation to the real world. 
Those who are familiar with maps can verify that the fact 
that maps are drawn with north being generally in the 
direction of the top of the map, is of little use when 
translating the information to the scene of interest. Regard- 
less of where north is, one tends to turn the map so that the 
direction ahead of the user, or in the direction of travel, in a 
real scene matches that direction on the map. This may result 
in the condition of an "upside down" map that is quite 
difficult to read (the case when the user is traveling south). 
Although translating the directions of the map to reality is a 
formidable task, it is an even greater problem to translate the 
symbols on the map to those objects in reality which they 
represent. The runnel symbol on the map does not show what 
the real tunnel actually looks like. The fact that the appear- 
ance of the tunnel from infinitely many points of view is 
prohibitively difficult to represent on a map accounts for the 
use of a simple symbol. Furthermore, the map does not have 
any indication from which point of view the user will first 
see the tunnel, nor any indication of the path which the user 
will take to approach the runnel. 

It is now possible to computerize city road map informa- 
tion and display the maps according to the path taken by a 
user. The map is updated in "real-time" according to the 
progress of the user through the city streets. It is therefore 
possible to relieve the problem of upside-down maps as the 
computer could re-draw the map with the text in correct 
orientation relative to the user even when one is traveling in 
a southerly direction. The computer generated map is dis- 
played at a monitor that can be easily refreshed with new 
information as the user progresses along his journey. Maps 
of this type for automobiles are well known in the art. Even 
very sophisticated maps with computer generated indicia to 
assist the user in decision making are available and 
described in patents such as DeJong U.S. Pat. No. 5,115,398. 
This device can display a local scene as it may appear and 
superimpose onto the scene, symbolic information that 
suggests an action to be taken by the user. For example, a left 
turn as is shown in FIG. 3 of the disclosure. Even in these 
advanced systems, a high level of translation is required of 



15,411 

2 

the user. The computer generated map does not attempt to 
present an accurate alignment of displayed images to the real 
object which they represent. 

Devices employing image supplementation are known 

5 and include Head Up Displays, HUDs and Helmet Mounted 
Displays HMDs. A HUD is a useful vision system which 
allows a user to view a real scene, usually through an optical 
image combiner such as a holographic mirror or a dichroic 
beamsplitter, and have superimposed thereon, navigational 

io information for example symbols of real or imaginary 
objects, vehicle speed and altitude data, et cetera. It is a 
primary goal of the HUD to maximize the time that the user 
is looking into the scene of interest. For a fighter pilot, 
looking at a display device located nearby on an instrument 

is panel, and changing the focus of ones' eyes to read that 
device, and to return to the scene of interest, requires a 
critically long time and could cause a fatal error. A HUD 
allows a fighter pilot to maintain continuous concentration 
on a scene at optical infinity while reading instruments that 

20 appear to the eye to also be located at optical infinity and 
thereby eliminating the need to refocus ones' eyes. A HUD 
allows a pilot to maintain a "head-up" position at all times. 
For the airline industry, HUDs have been used to land 
airplanes in low visibility conditions. HUDs are particularly 

25 useful in a landing situation where the boundaries of a 
runway are obscured in the pilots field of view by fog but 
artificial boundaries can be projected onto the optical com- 
biner of the HUD system to show where in the user's vision 
field the real runway boundaries are. The virtual runway 

30 projection is positioned in the vision field according to data 
generated by communication between a computer with and 
the airport instrument landing system, ILS which employs a 
VHF radio beam. The system provides the computer with 
two data figures. First a glide slope figure, and second, a 

35 localizer which is a lateral position figure. With these data, 
the computer is able to generate an optical image (photon) 
to be projected and combined with the real scene (photon) 
that passes through the combiner and thereby enhancing 
certain features of the real scene; for example runway 

40 boundaries. The positioning of the overlay depends on the 
accuracy of the airplane boresight being in alignment with 
the ILS beam and other physical limitations. The computer 
is not able to recognize images in the real scene and does not 
attempt to manipulate the real scene except for highlighting 

45 parts thereof. HUDs are particularly characterized in that 
they are an optical combination of two photon scenes. The 
combination being a first scene, one that is normally viewed 
by the users eyes passes through an optical combiner, and a 
second, computer generated photon image which is com- 

50 bined with the real image at an optical element. In a HUD 
device it is not possible for the computer to address objects 
of the real scene, for example to alter or delete them. The 
system only adds enhancement to a feature of the real image 
by drawing interesting features thereon. Finally, HUDs are 

55 very bulky and are typically mounted into an airplane or 
automobile and require a great deal of space and complex 
optics including holograms and specially designed lenses. 

Helmet Mounted Displays HMDs are similar to HUDs in 
that they also combine enhancement images with real scene 

60 photon images but they typically have very portable com- 
ponents. Micro CRTs and small combiners make the entire 
system helmet mountable. It is a complicated matter to align 
computer generated images to a real scene in relation to a 
fast moving helmet. HUDs can align the data generated 

65 image that is indexed to the slow moving airplane axis which 
moves slowly in relation to a runway. For this reason, HMDs 
generally display data that does not change with the pilots 
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head movements such as altitude and airspeed. HMDs suffer camera where the image displayed accurately represents the 

the same limitation as the HUDs in that they do not provide real scene as it appears from the point of view of one looking 

the capacity to remove or augment elements of the real along the optical axis. In this way, one easily orients the 

image. information displayed to the world as it exists. A fundamen- 

Another related concept that has resulted in a rapidly 5 tal difference between the Wsion system of the invention and 

developing field of computer assisted vision systems is m * 1 f a reorder «n be found in the image augmentatton. 

known as virtual reality, VR. Probably best embodied in the a m ^ the superposition of an 

fictional television program "Star Trek; The Next ^ data such as a "low battery indicator, et cetera, 

Generation", the "Holodeck" is a place where a user can go * has " 0 'knowledge of the scene that is being viewed. The 

to have all of his surroundings generated by a computer so 10 data displayed usually is related to the vision device or 

as to appear to the user to be another place or another place f ™thwg independent of the scene sucb as the time and 

and time e ' * ma S e augmentation of the invention can include 

,.. * „ , . . , information particular to a scene being viewed with the 

Virtual reality systems are useful in particular for a invention 

training means. For example in aircraft simulation devices. t, .'. 4 . . * , A . , 

a * j * •! * u • i ^ i is The vision system of the invention can include a data base 

A student pilot can be surrounded by a virtual cockpit ... , , • c j* 

, . . . r . „ / . • i with prerecorded information regardmg various scenes. The 

which is essentially a computer interface whereby the user - ... , t4 . , ril ~ . . , . ,. . . 

„- . „ , J x , . , . Ci precise position and attitude of the vision system indicates to 

reels the environment that may be present in a real aircraft, f, ,. r , , . . . . , A 

. , « t j • i 'the data base, the scene that is being viewed. A computer 

m a very real way and perhaps enhanced with computer . - . u . .u t 

\ . 3 . r j i_ • i ^ i- processor can receive information about the particular scene 

generated sounds, images and even mechamcal stimuli. t_ . . # , , . . r . _ 

~ . , i_ *. • • . t , , 4 . 20 from the data base and can then augment an image or the 

Actions taken by the user may be interpreted by the com- , , . * Tu • • * a 

, it J , J , : * At _ J ^ 4 scene generated by the camera of the vision system and 

puter and the computer can respond to those actions to f ~ , . J t iL « . iL . , 

r , , A . * , , 4 f present a final image at the display with includes a com bi- 

control the stimuli that surround the user VK machines can r 4 . r . r 4 . ° r__ a_ *• i ■ * j - c 

, , fc , . «• _ * nation of mtormation from the optical input and information 

create an entire visual scene and there is no effort to 4 . A 4 , . . , 4 , „ ^. , , . ^ 4 . 4 , 

. ^ . that was stored in the data base. Particularly important, is the 

superimpose a computer generated scene onto a real scene. ibili of comrnunicatioo between me ' dit!L V [lom the data 

A VR device generally does not have any communication £ , . A . . , 

: f , , • _, i* u - base and the real image. Analyzing and processing routines 

between its actual location in reality and the stimuli bemg . t , .7. r •* ■ £ 1 j 

• 1 « . t . ; , 4 , ™ 5 ma y include recognition of items in the real scene and 

presented to the user. The location of the VR machine and J . . , . - . , . t ~ . , , , 

*l . c . , . 4 . „ . comparisons with artifacts of the stored data. This could be 

the location of the scene bemg generated generally have no J r . . * . . , . . tU « , • . 

h<11 ... &6 & 7 useful in alignment of the real images to the recalled data, 

p ysic re a 10 lp. ^ In a s j tuat j on wnere tne optical input of a scene is entirely 

VR systems can be used to visualize things that do not yet b i ocke d from the camera of the system, for example by 

exist. For example, a home can be completely modeled with dense fog> an of the scene can be generated which 

a computer so that a potential buyer can "walkthrough" includes onlv information from the data base. Alternatively, 

before it is even builL The buyer could enter the VR ^ data base being of finite size may QOt have 

atmosphere and proceed through computer generated 35 mauon a b ou t a particular scene. In this case, the image 

images and stimuli that accurately represent what a home preS ented at the display would be entirely from the optical 

would be like once it is built. In this way, one could know of the real spcc \ a \ case reduces the vision 

if a particular style of home is likable before the large cost system t0 the equivalent of a simple camcorder or electronic 

of buHding the home is incurred. The VR machine being binocular. It is also a very special case where the features of 

entirely programmed with information from a designer does ^ a rea] scene are selectively removed. If the bright lights of 

not anticipate things that presently exist and there is no a c ity_ scap e obstruct the more subtle navigation lights of a 

communication between the elements presented in the VR marine ^ of entrVj ^ it is possib i e for me processing 

system to those elements existing in reality. routines to discriminate between the city lights and the 

While the systems and inventions of the prior art are navigation lights. The undesirable city lights could be 

designed to achieve particular goals, features, advantages, 45 removed in the processor before the final image is displayed, 

and objectives, some of those being no less than remarkable, In the final image, the important navigation lights show 

these systems and inventions have limitations and faults that clearly and the city lights are not present at all. Therefore, a 

prevent their use in ways that are only possible by way of the final image of the invention can be comprised of information 

present invention. The prior art systems and inventions can from two sources, in various combinations, superimposed 

not be used to realize the advantages and objectives of the 50 together to form a single, high information-density image, 

present invention. The information of the two sources are compared and 

combined together to form a single augmented image that is 

SUMMARY OF THE INVENTION presented at a display and is aligned to. the real scene as the 

Comes now, an invention of a vision system including scene is viewed by the user, 

devices and methods of augmented reality wherein an image 55 In the simplest form, devices of the invention can be 

of some real scene is altered by a computer processor to envisioned to include six major components: A 1) camera to 

include information from a data base having stored infor- collect optical information about a real scene and present 

mation of that scene in a storage location that is identified by that information as an electronic signal to; a 2) computer 

the real time position and attitude of the vision system. It is processor; a 3) device to measure the position of the camera; 

a primary function of the vision system of the invention, and 60 and a 4) device to measure the attitude of the camera 

a contrast to the prior art, to present augmented real images (direction of the optic axis), thus uniquely identifying the 

and data that is continuously aligned with the real scene as scene being viewed, and thus identifying a location in; a 5) 

that scene is naturally viewed by the user of the vision data base where information associated with various scenes 

system. An augmented image is one that represents a real is stored, the computer processor combines the data from the 

scene but has deletions, additions and supplements. The 65 camera and the data base and perfects a single image to be 

camera of the device has an optical axis which defines the presented at; a 6) display whose image is continuously 

direction of viewing as in a simple "camcorder" type video aligned to the real scene as it is viewed by the user. 
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The camera is related to the position measuring device 
and the attitude measuring device in that the measurements 
of position and attitude are made at the camera with respect 
to arbitrary references. The position and attitude measure- 
ment means are related to the data base in that the values of 
those measurements specify particular data base locations 
where particular image data are stored. We can think of the 
position and attitude measurements as defining the data base 
pointer of two orthogonal variables. The camera is related to 
the computer processor in that the image generated at the 
camera is an electronic image and is processed by the 
computer processor. The data base is related to the computer 
in that the data base furnishes the processor information 
including images for use in processing routines. The display 
is related to the computer as it receives the final processed 
image and converts the computers electric image signal into 
an optical image that can be viewed by the user. The display 
is boresight aligned with the optical axis of the camera such 
that the information corresponds to reality and appears to the 
user in a way that allows the user to view the final aug- 
mented image without needing to translate the image to the 
orientation of the real scene. 

In the simplest form, methods of the invention can be 
envisioned to include seven major steps: An 1) acquire step 
whereby the light from a scene is imaged by a lens; a 2) 
conversion step whereby optical information of the acquire 
step is converted into an electrical signal; a 3) position 
determining step in which the position of the camera is 
measured; an 4) attitude determining step in which the 
attitude of the camera is measured; a 5) data recall step 
where a data location is selected in accordance with the 
measurements in steps 3 and 4 and user impute data, is 
recalled by a computer processor; a 6) processing step 
wherein data from the data store and the electronic image are 
combined and processed; and a 7) display step wherein a 
processed final image is displayed. 

The product of the acquire step, an optical image, is 
converted to an electric signal in the conversion step. The 
electronic image of the conversion step is transmitted to the 
processor in the processing step. The products of the posi- 
tion determining step and attitude determining step are 
values that are used in the data recall step. The result of the 
data recall step is also transmitted to the processor to be 
combined with the electronic image of the conversion step 
in the processing step. The product of the processing step, a 
final electronic representation of an augmented image is 
transmitted to, and displayed in, optical format in the display 
step. 

The invention will be summarized further by presenting 
six examples of the invention wherein a description of the 
devices, methods, and uses thereof, follow. 

In a first summary example of the invention, the reader is 
to imagine a scenario where a boat is approaching a port of 
entry and the user of the invention is a navigation officer of 
that boat. It is quite common for a navigation officer to 
require many aids to guide his course through a shipping 
channel. Charts, a compass, lighted buoys, some devices, 
ranges, radar, binoculars are some of the instruments that 
one may use to navigate a boat. Recent advances in position 
determining technologies, in particular the Global Position- 
ing System, or GPS, have simplified the task of navigation. 
With the GPS, a navigation officer can rely on knowing the 
position of the craft to within approximately ±300 feet, north 
and east; and in some special cases within less. Even with 
such a good position determination, the navigation officer 
must locate where on the chart his position corresponds, and 
identify symbols on the chart to create a mental image of his 
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surroundings. Then the navigation officer must look about at 
the real scene before him for identifiable objects to deter- 
mine how what he sees corresponds to the symbols on the 
chart. Frequently, visibility is limited by darkness or weather 

5 conditions and particular lights must be recognized to iden- 
tify chart markings. These can be colored flashing lights and 
can easily be mistaken for the bright lights of a city skyline. 
In other cases, the markers may be un-lit and may be 
impossible to find in the dark. Dangerous objects, for 

io example sunken ships, kelp, and reefs, are generally marked 
on the chart but can not be seen by the navigation officer 
because they can be partially or entirely submerged. The 
navigation officer must imagine in his mind his position with 
respect to objects in the real scene and those on the chart and 

is must also imagine where in the real scene that the chart is 
warning of dangers. This procedure requires many complex 
translations and interpretations between the real scene, the 
markers of the chart, the scene as it is viewed by the 
navigation officer, and the chart as understood by the navi- 

20 gation officer. Obviously, there is great potential for mis- 
takes. Many very skilled and experienced naval navigators 
have failed the complicated task of safely navigating into a 
port resulting in tragic consequences. With the system of the 
invention, a navigation officer can look with certainty at a 

25 scene and locate exactly, known marks. The system of the 
invention eliminates the need for determining where in a real 
scene the symbols of a chart correspond. The user of the 
invention can position the display between his eyes and the 
real scene to see an image of the real scene with the symbols 

30 of a chart superimposed thereon. In the navigator* s mind it 
becomes very concrete where the otherwise invisible reefs 
are located and which lights are the real navigation lights 
and which lights are simply street lights. It is possible for the 
computer to remove information such as stray lights from 

35 the image as it is recorded from the real scene and to present 
only those lights that are used for navigation in the display. 
This is possible because the data base of the invention 
"knows*' of all navigation lights and the processor can 
eliminate any others. The display that the user views 

40 includes a representation of a scene with complicated unde- 
sirable objects removed and useful data and objects being 
added thereto. Whenever the navigator points the device of 
the invention in some direction, the device records the 
optical image of the real scene and simultaneously deter- 

45 mines the position and attitude of the device and calls on a 
data base for information regarding the scene being 
observed. The processor analyzes the image and any data 
recalled and combines them to form the final displayed 
image. 

50 In a further summary example of the invention, a city 
planning commission may wish to know how a proposed 
building may look in the skyline of the city. Of course it is 
possible to make a photograph of the skyline and to airbrush 
the proposed building into the photograph. This commonly 

55 used method has some shortfalls. It shows only a single 
perspective of the proposed building that may be presented 
in the "best light" by a biased developer (or "undesirable 
light" by a biased opponent/competitor). The building may 
be presented to appear very handsome next to the city hall 

60 as shown in the developer's rendition. Since only one 
perspective is generally shown in a photograph, it may be 
impossible to determine the full impact the building may 
have with respect to other points of view. It may not be clear 
from the prepared photograph that the beautiful bay view 

65 enjoyed by users of city hall would be blocked after the 
building is constructed. With the current invention, the 
details of every perspective could be easily visualized. Data 
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that accurately represents the proposed building could be 
entered into a data base of a device of the invention. When 
the camera of the invention is pointed in the direction of the 
new building, the camera portion records the real scene as in 
appears and transmits that signal to a processor. The device 5 
accurately determines the position and attitude of the camera 
with respect to the scene and recalls data from the data base 
that properly represents the perspective of the building from 
that point of view. The processor then combines the real 
scene with the data of the proposed building to create a final 10 
image of the building from that particular perspective. It 
would even be possible for a helicopter to fly in a circle 
around the location of the building and for a user to see it 
from all possible points of view. A council member could see 
what the future structure would be like in real life from any !$ 
perspective before voting to approve the plan. 

In a still further summary example, we choose a scenario 
where an engineer uses products of the invention for analy- 
sis and troubleshooting of an engineering problem. In 
particular, the case where a problem has been detected in the 20 
plumbing aboard a submarine. The complicated works 
including pipes, tubes, pumps, cables, wires, et cetera, of a 
submarine may be extremely difficult to understand by 
looking at a design plan and translating the information from 
the plan to the real world. Immediate and positive identifi- 2s 
cation of a particular element may be critical to survival of 
the ship in an emergency. The following illustrative engi- 
neering use of the invention provides for a greatly simplified 
way of positive identification of engineering features. 

An engineer aboard a submarine is tasked to work in the 30 
torpedo room on a saltwater pump used to pump down the 
torpedo tubes after use. In preparation for the job, the data 
base of a portable electro-optic vision device is updated 
from the ship's central computers with information regard- 
ing the details of the pump in question and of the details of 35 
the torpedo room where the pump is located. In the event of 
a battle damaged ship or in case of limited visibility due to 
fire or power outages, the vision device can provide guid- 
ance to the location of the pump through visual and audio 
clues. As the various pipes may be routed through bulkheads 40 
and behind walls, the vision system can be used to see 
through walls and to "fill-in" the locations of the pipes so 
that the otherwise hidden pipe can be followed continuously 
and without ambiguity. Upon arriving at the pump in 
question, the engineer points the camera axis of the vision 45 
device in the direction of the pump. In the display of the 
device, a real image of the pump is superimposed with data 
such as the part number of the pump and clues to features of 
the pump such as the type of material being pumped and 
flow direction. The vision device can receive inputs from the 50 
user to select advance display modes. This input may be 
made by way of mechanical devices like a "mouse", or a 
simple button, or may be verbal commands that are recog- 
nized electronically by speech recognition means. If the 
pump is to be disassembled, the user may be able to instruct 55 
the device to display sequential disassembly steps by a 
verbal "step" command. Disassembly could then be simpli- 
fied by clues provided by the vision system's display. A 
particular part can be highlighted and the motion required 
for its proper removal can be simulated such that the user 60 
can learn the correct disassembly procedure in real time. 
Similarly, re-assembly can be expedited because prior 
knowledge of the reassembly sequence rests in the data base 
and is easily presented to the user in a way that does not 
require translation from a parts book to reality. A highlighted 65 
part and the motion to assemble that part can be superim- 
posed onto a real image of the workpiece. 
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In a still further summary example, one can imagine the 
complexity of a situation where a structure fire is being 
fought. It may be typical for a fire captain to be reading the 
detailed features of the building interior from a blueprint and 
to transmit that information by radio to firefighters on the 
inside of the building. The problem with this method is that 
it depends on multiple translations between various media. 
With each translation, the information is distorted according 
to some transfer function thus limiting its accuracy. The 
present invention offers an alternative which allows detailed 
information of a structure to be directly displayed to a 
firefighter inside the building. 

A video camera and goggle set could be installed into a 
standard firefighter helmet, the goggle set having therein, the 
requisite display. The display unit would be in 
communication, either by hard wire or radio, to a computer 
processor that is also in communication with a data store 
having information prerecorded therein regarding features 
of the structure. A locating system could determine at all 
times where the firefighter is positioned and where she is 
looking. A real image from the camera can then be combined 
with the recorded data to realize a final augmented image. If 
the bright light generated by the fire in the real image is 
blocking important features such as a door handle, the 
processor can delete the fire from the scene and "draw-in" 
the door handle. The firefighter could easily navigate her 
way through a complicated building interior full of smoke 
and fire that would otherwise prohibit ones progress through 
that building. First hand information provided by a vision 
system of the invention directly to a firewoman can greatly 
increase the speed and effectiveness of fire fighting tasks. A 
device mounted into a helmet would necessarily be small 
and compact. It may be arranged such that the camera, 
display, and the locating (position and attitude) means are in 
radio communication with a computer and processing means 
that may be bulky but located remotely. This possibility is 
considered to be a subset of, and included within, the scope 
of the invention. 

It should be pointed out that in the case of very heavy 
smoke where no useful real image can be detected by the 
camera and the displayed image is consists entirely of data 
from the data store, the system becomes quite similar to the 
virtual reality devices. The critical distinction is that the 
images displayed are exactly aligned to reality and oriented 
to the direction in which the user is looking. If the firefighter 
pushes on a door, a real door will open. In VR applications, 
it will appear and feel like a real door is being opened, but 
the door of a VR system is completely simulated and there 
is no real door being opened. 

In a still further summary example, to provide a viewing 
device that allows a tourist to "see" into a previous time. One 
can imagine a bus load of tourists arriving at he grounds of 
Pompeii. Each person equipped with portable vision system 
could be free to roam the grounds at will. It is possible to 
have predetermined points where the user could be directed 
and to have a sound track that corresponds to the scenes as 
presented to the user. Looking into the display of the vision 
system a user can see the buildings as they are now and a 
simulation of how they appeared prior to the eruption of Mt. 
Etna. He sees a simulation of people going about their daily 
fives in ancient Pompeii. Without warning, the viewer sees 
the ash falling on the buildings and people running for cover. 
The simulated events of the past can be easily viewed in a 
modem time with provisions of the present invention. In this 
version, each portable device can be in radio communication 
with a shared data and processing center that communicates 
with each individual personal device independently. 
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la a final summary example of the invention, we consider 
the need for surveyors to "see" into the ground to locate 
pipes and wires that have been buried. When construction 
projects require that such pipes and wires be removed or 
altered, or new projects to be installed such that they do not 
interfere with existing works, then surveyors must attempt to 
locate those works. A surveyor relies on documentation of 
the previous construction projects and attempts to plot onto 
the real world the locations of the various works as indicated 
by those maps. Again, the surveyors translation from maps 
and other documentation to the real world and to the 
operators of the digging equipment require skill and expe- 
rience. Even with recent and accurate maps, the task can 
become a nightmare and failures very expensive. If one 
could look at a display aligned to the real scene which 
included an image of the scene and superimposed thereon 
the positions of the buried works, then it would be a trivial 
matter to mark the locations for caution. With the system of 
the invention, the buried pipes, and wires, et cetera, could be 
recorded in the data base. The locations of construction 
projects could then be surveyed in a very short time. 

It should be noted that most mapping systems of the art, 
even very sophisticated computerized maps with data and 
data symbols combined with images, require a translation of 
the presented image to the images of the real world as 
viewed by a user. Although HUDs combine a real image and 
data images that are aligned to reality, the invention can be 
distinguished in that it has the capacity for communication 
between the real image and the data or data images. They 
cannot provide for recognition, alignment, undesirable fea- 
ture extraction, and others advanced features of the inven- 
tion. Only the present invention provides a system of pro- 
ducing augmented images aligned to a scene as the scene is 
naturally viewed. 

It is a primary object of the invention to provide a 
versatile vision system having capabilities completely 
unknown to systems in the art. 

It is a further primary object of the invention to introduce 
to surveying, engineering, touring, navigation and other arts, 
a device and method of augmented reality whereby a real 
scene is imaged with augmentation which provides addi- 
tional information about the scene to the user. 

It is a further object of the invention to provide a vision 
system for seeing a topography that includes objects that do 
not exist at the time of use, but are intended to be a part of 
the scene in some future time. 

It is a further object of the invention to provide a vision 
system for seeing a topography that includes objects that can 
not be seen in conventional viewing systems including 
cameras, binoculars, telescopes, video recorders, head up 
displays, helmet mounted displays, et cetera. 

It is a still further object of the invention to provide a 
vision system that allows undesirable objects of a real scene 
to be removed and replaced with computer generated objects 
to represent objects that are important to the user. 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other features, aspects, and advantages of the 
present invention will become better understood with regard 
to the following description, appended claims and drawings 
where: 

figure one is a drawing of a user of the invention, a scene 
as it appears in reality, and the scene as it appears in the 
augmented display of the device of the invention; 

figure two is a drawing of a city street scenario (2a) 
whereby the invention allows a surveyor to see otherwise 
invisible features such as buried pipes and sewer lines (2b); 
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figure three is a block diagram arrangement of compo- 
nents and their relationship to each other that sets forth the 
novel combination of the invention; 

figure four is a further detailed block diagram; 
5 figure five is a electro-optical schematic drawing of some 
of the system components. 

PREFERRED EMBODIMENTS OF THE 
INVENTION 

10 

The invention provides for a vision system device oper- 
able for producing an augmented image. In drawing figure 
one, a user 2 is shown to be looking into a canyon through 
a portable assembly 1 of the invention. An augmented image 

15 3 is presented to the user and is aligned with the real scene. 
The augmented image may include: elements that do not yet 
exist, for example a bridge 4, representing information 
added to a real scene; elements that exist in the real scene but 
do not appear in the final image, for example trees at the 

20 right side of the bridge, representing information that has 
been deleted; and elements that have been changed from 
their actual appearance. In drawing figure two, the example 
of construction site surveying is illustrated. A scene as 
viewed normally would look like figure two(a). In figure 

25 two(b), the fresh water pipes 5 leading to homes 7 and fire 
hydrants 8 and the buried sewer lines can be easily "seen" 
in the display of the invention. 

In a first preferred embodiment of the invention, an 
electro-optic apparatus comprising a camera 9, a position 

30 determining means 16, an attitude detennining means 15, a 
data store 12, a computer 14, and a display 13; the camera 
being an electro-optic device operable for converting a 
photon input from a field of view, into an optical image at 
an image field and operable for converting that photon 

35 image into an electronic image and operable for transmitting 
that electronic image to the computer, the positioning means 
being operable for determining the position of the camera, 
the attitude means being operable for determining the cam- 
era pointing attitude of the camera defined by the symmetry 

40 axis of the camera field of view, the data store being in 
communication with the position and attitude determining 
means whereby the values of position and attitude indicate 
a location or locations in the data store being further in 
communication 39 with the computer whereby image data 

45 can be transmitted thereto, the computer whereby the elec- 
tronic image from the camera and the electronic image 4 of 
the data store are compared, analyzed, processed and com- 
bined for display as a single image, the computer being in 
communication with, the display, whereby camera position 

50 and attitude information particular to the scene are used to 
augment a real image of a scene that is particular to the 
position and attitude of the camera, is provided. A detailed 
description of each of the main device elements follows, 
camera 

55 In preferred embodiments of the invention it is anticipated 
that the camera portion of the device will be comprised of 
four main subsystems integrated together into a portable 
assembly 1 that is easy for a user 2 to hold. These include: 
an optical input lens assembly 38, an image stabilization 

60 system, at least one charge coupled device, CCD 20, and an 
optical ranging system 34. 

A photon input imaging lens assembly can be mounted 
into the portable camera unit 1. Single axis, or "monocular" 
systems are commercially available and are commonly used 

65 in devices such as camcorders and digital photographic 
cameras. It is preferred that the lens have a zoom and 
autofocus capability; these systems are well developed in the 
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art. An autofocus system can be computer driven but releas- 
able for user override. Lens zoom operations can also be 
power driven and controlled manually by the user at an 
electronic control switch, or could be driven by the computer 
based on known features particular to a chosen scene. The 5 
physical dimensions of the input lens define the field of view 
of the device which is variable with the zoom properties of 
the lens. Information regarding the field of view can be 
supplied to the processor to be used in image combination 
routines. 10 

Image stabilization systems are also well developed in the 
electronic imaging arts and the known systems are useful for 
preferred embodiments of this invention. Image stabilization 
can be realized by way of solid state piezo-electric devices 
to drive a distortable prisms. Motion information from 15 
sensitive gyros 35 could detect movements and provide a 
driving signal to the prisms. The system of the invention can 
be modified so that output from each of the gyros is made 
available to the control computer for calculations of the 
attitude and position of the portable assembly. This can be 20 
particularly important for systems not having access to a 
GPS signal, for example in a submarine or in a system 
designed for microscale applications such as surgery or 
micro manipulation. As an alternative to mechanical stabi- 
lization systems, an electronic image stabilization system 25 
could be used but there could be a loss of image quality and, 
in certain situations of rapid motion, a loss of continuity in 
image input. Also, an electronic image stabilization system 
could not be readily modified to provide motion information 
to the control computer. 30 

After the photon optical input from a scene is stabilized by 
the image stabilization system, it is then focused onto at least 
one charged coupled device, CCD, that is positioned in the 
image plane of the input lens. A possible alternative could be 
a silicon intensified camera or other image-con device but 35 
CCDs are preferred because their output can be compatible 
with electronic computers using digital data. The light field 
from the scene is converted to a digital electronic image 
signal which represents the scene. CCD devices are also 
quite commonly available and used in modern imaging 40 
devices and standard CCDs are sufficient for use in the 
present invention without modification or with only minor 
changes. The camera can be color sensitive and two com- 
mon methods for producing color images are possible. It is 
also anticipated that CCD devices that are sensitive to 45 
portions of the spectrum other than the human-visible region 
may be useful. For example some systems may benefit from 
CCD devices that are tuned for infrared IR imaging. Use of 
IR devices may be important for applications such as fire 
fighting and agronomy where the important image infonna- 50 
tion may be in a spectrum other that the visible region. 

A ranging system is useful for a plurality of system 
functions. A first system function is to provide for the focus 
of the input lens. For images to be properly focused onto the 
CCD, it is necessary to adjust the input lens according to the 55 
distance of the objects being imaged. As a second system 
function, range information is also useful in the processing 
stage as it gives hints to the proper scaling of the real image 
with respect to the data from the data store. The camera 
image and data store inputs will be combined such that scale 60 
is an important consideration for proper appearance of the 
final image. A further use, a third use of optical ranging can 
be to eliminate unwanted image features. For example, if a 
ship's rigging tends to block the view of the shore, then it is 
possible to input to the image processor the range that is of 65 
interest. The processor can then remove objects that are not 
within the selected range. 
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A ranging system, for example a sonic or ultrasound 
device, can measure the distance from the camera to the 
objects of a scene; the range. It is also possible to use laser 
or radar systems but they tend to be more complicated and 
more expensive. The computer can receive range informa- 
tion directly from a ranging system or can receive range 
information from the focus position and zoom condition 18 
of the input lens that can be interrogated by a computer 
routine. This is possible if the lens is designed with trans- 
ducer devices that convert the focus and zoom lens condi- 
tions into electronic signals that are readable by the com- 
puter. As an alternative to active ranging systems or as a 
supplement to, it is possible to determine the range to objects 
of a scene from comparisons of image features to known 
elements in the data base that correspond to those image 
features. A disclosure of this method is presented in U.S. Pat. 
No. 5,034,812, Rawlings. Once position and attitude of the 
portable camera is established, a real image may be input 
and analyzed by the computer. By reference to a comput- 
erized topographic data base, the position and therefore 
distance and bearing to user selected features may be 
established. An example; an object such as a lighthouse 
whose height was previously stored in the data base may be 
measured from the real image where the magnification 
conditions of the lens are known thereby making a range 
determination possible. Each of the ranging measurements 
may be made independently or in combination with a 
plurality of other measurements to increase accuracy, 
position determining means 

In preferred embodiments of the invention it is anticipated 
that the position determining means of the device will be a 
Global Positioning System GPS receiver 16. There are many 
alternative positioning systems that could be used effectively 
for other embodiments. These include, but are not limited to, 
Loran, Glonass, and Omega. Since GPS has very high 
accuracy and has altitude capability, it is an obvious choice 
for the invention which benefits from the extra precision of 
that system. In self contained environments like subterra- 
nean or a submarine, the positioning means may have to be 
a simplified version of triangulation from known locations 
or other positioning determination means. For applications 
on a microscale, for example semiconductor inspection or 
microdevice manufacture, it may be useful to have a laser 
interferometer position measurement means that has accu- 
racy at the sub -nanometer level. The invention does not 
depend on the particulars of any positioning means just so 
that the position can be determined and input into a com- 
puter processor. 

One objective of the invention is to have a small, 
lightweight, portable camera. To achieve this, it may be 
necessary to put the bulk of the computing power in a 
separate unit that is in communication with the portable 
camera. In this embodiment, the antenna 36 for the GPS 
receiver would be preferably put within the portable camera 
or in close proximity thereto, and the more bulky processing 
portion of the GPS receiver could be combined with other 
system computing facilities in a backpack or ground unit. 
For units to be used in obstructed locations where a GPS 
signal may not be available such as within the bridge of a 
ship, the antenna can be placed in some known position 
relative to the portable camera and the known and constant 
displacement therebetween removed in the processing elec- 
tronics. 

attitude determining means 

In preferred embodiments of the invention it is anticipated 
that the attitude determining means of the device will be one 
of three alternatives. Each provides the heading, tilt and roll 
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of the portable camera unit. A flux gate compass 37 located matrix LCD 32 device. These are currently the brightest, 

within the portable camera provides heading information to fastest displays available and are quite commonly used in 

an accuracy of ±1 degree. The output from this flux gate conjunction with video applications. Alternatively, it is 

compass can be made available to the computer. Outputs possible to use plasma display, electroluminescent, or any of 

from piezo-electric gyros located in the image stabilization 5 several possible displays. The particular type of display is 

system can also be used by the computer to calculate motion not important and may be application specific without 

from a datum established by the user. The datum can be from deviating from the gist of the invention. An important 

rest in a known level position or can be established by feature of the d[ { fe that an electronic signal generated 

observing the honzon by looking through the portable from a uter video card ^ can ^ conV erted into a 

camera. An alternative is a tmxial magnetometer system 1Q yM[e ^ f ^ ^ aiQd real 

mcorporating a biaxial electrolytic inclinometer. This trans- * addressing. It is also 

ducer is located within the camera and is operable for 6 . c - . 4 . u .u r • . j 

providing the computer with a complete attitude determina- a m ? 0T feature of the uivention to have the display oriented 

iion. It is a further alternative to compute attitude (including ™f h re /^ and lherefore ^^ ed with respect to the optical 

heading) from a known datum using the piezo-electric axis of the camera as 15 shown m nG - 1 

gyros. Selection of this alternative may be for reasons of 15 otner 

lowered cost as well as permitting application where mag- Io preferred embodiments of the invention it is anticipated 

netic fields may preclude accurate heading readings ( i.e. that the device can also include such apparatuses such as 

close to electric arc welding equipment; large transformers user input control keys 29 to interact with the computer 

and etc.) Note that since the image stabilization system routines and to specify information that drives further com- 

already contains the required piezo-electric gyros, it would 20 puter routines; audio transducers 31 both input types such as 

be possible and desirable to provide this alternative as an microphones and output types such as speakers also for 

optional method for establishing headings in all above control of computer routines and for presentation of infor- 

alternatives. computer mation to a user. Standard equipment known to couple 

In preferred embodiments of the invention it is anticipated vision devices to human users and physical conditions such 

that the computer processor of the device will be a micro- 25 as a tripods, lens shades, eyepiece fixtures 30, et cetera are 

computer with very fast graphic and video abilities. Because all considered to be compatible with the device of the 

of the advanced state of computer development, many of the invention. 

necessary elements of a computer, for example fast frame The invention also provides for methods of producing an 

grabbers 22 and massive cache memories 23, have already augmented image. 

been developed. The computers and software used in video 30 In preferred embodiments of the invention, a vision 

games and computer generated animation applications lend system method comprising: an acquire step, a conversion 

themselves to the tasks at hand and can be readily be step, a position determining step, an attitude determining 

converted to those tasks. As the functions of the present step, a data recall step, a processing step, and a display step; 

invention are very unique, the necessary software to achieve wherein the imaging step the objects of a scene are imaged 

those functions will also necessarily be unique. It is antici- 35 onto a two dimensional plane as the light emanating from the 

pated that the complicated system instruction set design, objects propagates through at least one lens and forms the 

either implemented in hardware such as ROM or in image at the plane, wherein the digitizing step the two 

software, will be proprietary. It is likely to have task specific dimensional optical input is converted into an electrical 

instruction sets and to have many devices of the invention signal, wherein the position determining step the position of 

each having their own particular instruction set 40 the camera is determined with respect to some arbitrary 

data store reference point, wherein the attitude determining step the 

In preferred embodiments of the invention it is anticipated attitude of the camera is determined with respect to some 

that the data store means of the device will be a mass arbitrary direction, wherein the data recall step data from a 

memory device; for example a CD-ROM. The data store can data base is recalled in accordance with the measurements of 

have pre-programmed information regarding particular 45 the previous two steps, wherein the processing step the 

scenes that are of interest to a specific user. The position image of the digitizing step and the data of the data recall 

determining means and the attitude determining means step are combined to form a final image, and wherein the 

control the pointer of the data store. In the simplest display step the final electronic image is converted to an 

embodiment, the data store has locations defined by an optical image and displayed such that it is aligned to and 

orthogonal array of two variables. The values of position P M 50 corresponds with the real scene, is provided. A detailed 

and of attitude A„ uniquely define a corresponding scene description of each of the main method steps follows. 

SCENE{P„, A„}, In advanced versions, the range can also be the acquire step 

important and a range variable, a third orthogonal variable In the acquire step, a camera of the device is pointed in the 

defines a three dimensional array: SCENE{P„, A^, R„} The direction of a scene and a photon image is formed by a lens, 

variable defined by R tells the computer in which plane 55 The photon image is formed onto an electronic device that 

normal to the axis of the camera, lies the information of has been placed into the image plane of the camera lens. The 

interest. In even further advanced versions, a magnification desired field of view can be changed with the zoom function 

variable M„ can be similarly used to give a four dimensional of the lens to selectively choose portions of a scene. The 

array: SCENE{P„, A n , R„, M„}. Many other factors can be acquire step is likely to be continuous in preferred embodi- 

used to control the way that data is stored and recalled. Each 60 ments and pointing at various objects will result in a 

of the methods being particular to the task at hand, are continuously updated display. Real-time functions such as 

subsets of the general case where data regarding a scene is "panning" in which many acquire steps occurs in rapid 

selected from a data store in accordance to identification of succession, are completely anticipated. The acquire step can 

the scene being viewed. be user influenced. If a pre-selected range is chosen by way 

display 65 of a user input, the computer can accommodate and adjust 

In preferred embodiments of the invention it is anticipated the lens in accordance with that input. This becomes more 

that the display 13 means of the device is a common active clear in the example where the user wishes to see the 
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features of a distant object but the automatic ranging is 
keying on some object in the foreground. A selected range 
can eliminate acquisition of undesirable foreground objects, 
the converting step 

The method of the invention distinguishes itself from 5 
methods such as HUDs and HMDs in that a photon repre- 
sentation of a scene is converted into an electronic signal 
that can be processed by a digital computer. The image of the 
real scene can be converted into image frames at standard 
video rates. In this way, changes are not required to existing 10 
video equipment and that equipment can be easily integrated 
into systems of the invention. The digital signal of a CCD is 
particularly desirable because a great deal of processing 
hardware compatible with CCDs already exists in the art. 
The pixel representation of images of CCD devices is also 15 
compatible with the pixel representation of images of LCD 
displays. The digital image of the CCD can be directly 
transmitted to the computer processor routines, 
the determine position step 

The position of the camera can be determined in global 20 
(macro) applications with GPS methods and in microscopic 
applications with a laser interferometer. The positioning step 
is used to locate the camera with respect to an arbitrary point 
and to relay the result of that measurement to the computer 
such that the computer can identify where in the data base 25 
the information that corresponds to the scene that the camera 
is addressing is located, 
the determine attitude step 

The attitude of the camera optical axis with respect to all 
three axis of rotation is measured to identify the direction the 30 
camera is pointed thereby further identifying what scene the 
camera is addressing and what appears in the camera field of 
view. The attitude figure typically drives one of the data 
store pointer variables and thereby, when in combination 
with a position figure, uniquely selects data stored therein. 35 
Attitude can be realized differently in various applications. 
In marine applications, "heading" and an implied horizon 
can uniquely define a scene. This is considered a special case 
of attitude on all three axis of rotation. It many applications, 
altitude will need to be considered for a scene to be properly 40 
identified. The primary function of the position and attitude 
steps are to give the computer enough information to unam- 
biguously identify a scene, 
the recall recorded data step 

Having made a position and attitude determination, the 45 
minimum amount of information needed to specify a loca- 
tion in the data base is realized. For some simple 
applications, this is enough information to recall stored data 
that can be combined with a real image produced by a 
camera looking into a scene located by the position and 50 
attitude measurements. In advanced applications, other fac- 
tors such as range to object and magnification, may also 
need to be specified to specie the data to be recalled. The 
data after being recalled is transmitted to the processor to be 
combined with real image data. 55 
the processing step 

Real image data are received into the processor from the 
converting step and augmentation data is received into the 
processor from the data recall step. Feature recognition 
techniques and image manipulation routines are applied to 60 
two forms of information to finally yield an image that is a 
desired combination of the two. Certain undesirable infor- 
mation can be removed from the image of the real scene. 
Important information not appearing in the image of the real 
scene, but found in the data can be enhanced in the final 65 
image. 

display augmented image 
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A augmented real image comprised of real image infor- 
mation and data from the data store is assembled in the 
processor and transmitted in standard video format to a 
display device. The final image is then displayed in accurate 
alignment with the real scene in a way that allows the viewer 
to "see" objects and features, otherwise invisible in other 
vision systems, in perfect orientation with the real scene. 
Augmented real images can be generated at video rates 
giving a display that provides a real-time view of reality with 
computer generated augmentation combined therewith. 

Although the present invention has been described in 
considerable detail with reference to certain preferred ver- 
sions thereof, other versions are possible. Therefore, the 
spirit and scope of the appended claims should not be limited 
to the description of the versions contained herein. 

In accordance with each of the preferred embodiments of 
the invention, there is provided an apparatus for and method 
of realizing a vision system. It will be appreciated that each 
of the embodiments described include both an apparatus and 
method, and that the apparatus and method of one preferred 
embodiment may be different than the apparatus and method 
of another embodiment. 

What is claimed is: 

1. An electro-optic apparatus for producing an augmented 
image from an image of a real scene and a computer 
generated image, the augmented image being aligned to the 
scene such that the directions up, down, left and right of the 
augmented image correspond directly with up, down, left 
and right of the real scene and the normal direction of the 
image plane points in a direction away from the user and 
towards the scene, said augmented image being comprised 
of information generated by an imaging means and infor- 
mation recalled from a data store, the information being 
particular to the position as determined by the apparatus and 
attitude of tbe apparatus, which comprises: 

an imaging means; 

a position determining means; 

an attitude determining means; 

a data store; 

a computer; and 

a display, 

said imaging means having an optic axis which is a 
symmetry axis and defines the direction of viewing, 
a lens symmetric about that axis which defines an 
image field, and a charge coupled device in the 
image field; 

said position determining means having a reference 
point corresponding to said imaging means; 

said attitude determining means having a reference 
direction parallel to the direction of viewing of said 
imaging means; 

said data store having memory locations wherein pre- 
recorded data are stored, a plurality of orthogonal 
variables which identify those memory locations, 
and a pointer responsive to values from the position 
and attitude determining means which selects par- 
ticular memory locations containing particular pre- 
recorded data corresponding to the real scene; 

said computer being in electronic communication with 
said imaging means, position and attitude determin- 
ing means, data base, and display, having graphics 
and image processing capabilities; and 

said display being an elecro-optic emissive display 
having a normal direction orthogonal to the display 
plane, the normal direction being colinear with the 
symmetry axis of the imaging means and being in 
communication with said computer. 
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2. The apparatus in claim 1 where the display is integrated 
into a binocular type viewing device having an optical path 
which corresponds to each of two eyes. 

3. The apparatus of claim 1 further comprising an optics 
system of lenses to image the display at optical infinity to 5 
give the user the appearance and feeling of actually looking 

at the real scene as opposed to looking at a display device in 
the near field thereby allowing the user to relax the eye 
muscles by focusing at infinity. 

4. The apparatus of claim 1 said computer being respon- 10 
sive to image features of the real scene as indicated by some 
predetermined condition. 

5. An electro-optic apparatus for producing an augmented 
image from an image of a real scene and a computer 
generated image, the augmented image being aligned to the 15 
scene such that the directions up, down, left and right of the 
augmented image correspond directly with up, down, left 
and right of the real scene and the normal direction of the 
image plane points in a direction away from the user and 
towards the scene, said augmented image being comprised 20 
of information of generated by an imaging means and 
information recalled from a data store, the information being 
particular to the position as determined by the apparatus and 
attitude of the apparatus, which comprises: 

an imaging means; 25 
a position determining means; 
an attitude determining means; 
a data store; 



a computer; and 
a display, 



30 
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said imaging means being sensitive to photon input and 
operable for converting light into an electronic signal 
processable by said computer; 

said positioning determining means being operable for 
determining the location of the imaging means and 
presents a value which represents that location to the 
computer; 

said attitude determining means being operable for 
determining the attitude of the imaging means as 
defined by the axis of the imaging means and pre- 
sents a value which represents that attitude to the 
computer; 

said data store being operable for storing prerecorded 
data corresponding to and representing objects 
known to be in the present field of view of the 
imaging means as defined by the present position and 
attitude thereof, recalling that data and transmitting 
it to the computer; 

said computer being operable for receiving signals 
from imaging means and data store and combining 
those signals such that a composite image signal is 
formed and further operable for transmitting the 
composite image to said display; 

said display being an elecro-optic emissive display 
operable for converting the composite image signal 
of the computer to a physical signal viewable by a 
user that is aligned to the viewing axis of said 
imaging means, the electro-optic vision apparatus 
allows the user to see his environment with computer 
generated augmented images aligned to the scene as 
the user would view the scene normally. 



04/01/2004, EAST Version: 1.4.1 



US 6,381,583 Bl 

11 12 

(e.g., an instruction for the person filling the order regarding The foregoing defines the preferred embodiment of a 

what to do if a particular selection is not available); entering computer-implemented method of simulating movement of 

a note can be by any suitable means of the system, such as a shopper through a shopping facility to enable the indi- 

by entering text through the keyboard of the computer 20. vidua! to observe, inspect and select a product in the 

Once a list has been created for the products to be 5 shopping facility through operation of a computer. Stated 
purchased, the products can be ordered electronically another way, this is a method of creating a virtual shopping 
through ^Internet system with instructions for delivery by facility for interactive shopping by computer, 
the store or for pickup by the shopper going to the store. Based on the system described above, this method corn- 
Payment provisions (preferably secure ones) can also be prises selecting a physically embodied shopping facility 
included in the system. 10 having displays of goods physically disposed therein for the 

If the shopper wants to go to the store and go through the particular implementation of a grocery store, for example, 

aisles to select the individual items himself or herself, the The selected shopping facility has a known configuration via 

desired list can be printed (e.g., by selecting the "print" its actual construction and arrangement of products dis- 

button on any of lists 42, 44, 46) and taken to the store by played at the selected shopping facility such that a shopper 

the shopper. 15 can move among the displayed products to observe, inspect 

Returning to the home page 24 shown in FIG. 3, one or aDd from amon S the Splayed products, 

more list forms is displayed at the respective computer 20 if The method simulates the configuration of the displayed 

the shopper selects "make list." These can be of the same products in a computer, such as one of the computers 18 or 

type shown in FIGS. 5 and 9 and can allow the shopper 20. This includes in the illustrated implementation translat- 

either to create a current list or to install a reorder list. ing the known configuration of displayed products into 

Referring to FIG. 10A, to make a current list, the shopper encoded configuration-defining electric signals in memory 
enters the current date, selects any reorder items from an of the computer 18 and using the electric signals to create 
existing reorder list, enters new items to be selected and then vldeo representations of the configuration of displayed prod- 
controls the computer 20 to automatically organize in the „ ucts through the computer 20 (m another non-limiting 
manner the shopper wants (e.g., by location in the store or example, the signals can be stored on a storage medium used 
by price or alphabetically as illustrated in FIG. 9) and print directl y m the computer 20). Conversion of the images of the 
the list if desired. The shopper can then end the session or Physically disposed displays of goods into encoded digital 
"go shopping" to see if additional items are needed. sl S nals ™ by means of the digital camera 8 for the particular 

, j ,. . . , , . implementation of FIG. 2. The encoded digital signals from 

To make a reorder list, the shopper enters the current date, 30 . * a t . . f D . f. . . 

4 , . 4 4 , 1 1 j . - _r 4l _ the camera 8 are stored in the computer 18 in this ltnple- 

the item to be reordered and information regarding the 4 . , , . r _ . , r , 

1 n j*_*u jt*- ■ mentation. Access is provided to these stored signals such 

reorder frequency, and stores the reorder list in memory in . A . 1 . i_i * • -t_i 

* m ( l « 10 T*. i- • u that portions of the signals are selectable and transmissible 

either the computer 20 or the computer 18. The list can be . r „ j t . . - , 

• * j j »u u *l u u • » * ji to the computer 20 for displaying, on a monitor of that 

printed, and the shopper can then go shopping or end the ■ ^ 1 \ % *i_ j • 1 c j 

r -..J a 1 r 1* 4 7 *i_ j computer, virtual movement alone the display of goods 

session as desired. An example of a resultant output based on 35 r . . , j , . • f ■ 1 7 . 

... t r,.- 1 - T-Ti-i c u i- . -ift u- u u represented by selected stored encoded digital signals. In the 

this type of list is shown in FIG. 5 by list 30, which shows ... r A , . ' ..... 1 T . * 

f . a a t *u \a. c u r * m illustrated implementation this uses known Internet commu- 

items to be reordered as of the current date. Such a list 30 . . , y t _ . t _ . 

u-i^ju^ur **u a * a nication technology tor accessing and transferring the req- 

can be defined by the foregoing to have a predetermined . . . bJ & & 1 

order frequency or a list can be automatically maintained um e 

based on actual ordering history (e.g., each time an actual 40 ^ shopper using the system and method of the present 

order list is made, the historical list updates and determines invention is located at a respective one of the computers 20 

ordering frequency such as by computing an average time and indicates to that computer 20 a first virtual position of 

between purchases of the same product). me shopper within the displayed shopping facility. This first 

tU . j u *u u **A <*™~ * • position is indicated by the shopper controlling the control 

The third option shown on the home page 24 of FIG. 3 is y ^ A , J f*. 4 , 6 , 

*u « j * 1 * »» r"\ * *i c *l* t. • means 14, such as a mouse, to place the cursor 34 at a 
to use the product locator. Details of this are shown in 45 , . . ' . ... , . r 
t-i^ mxA j* a a' * j. deseed location withm the displayed video representation of 
FIG. 10A- A predetermined directory of all the products . _ . . t r ' , r 
available from the store 2 is encoded in digital signals stored me configuration of displayed products, 
in the computer data base of the system such that this ^ sho PP er indicates to the computer a continuous 
directory is searchable by one or more words entered into the change from the first position to a second position. This can 
computer 20 by the shopper (or other selection technique, 50 be done bv moving the cursor 34 along the displayed image, 
e.g., an alphabetical list the shopper scrolls through). The In response the stored data within the predetermined display 
programmed computer 20 with its keyboard (for example) ran ge of the cursor location is acquired to continuously 
provide means for enabling the shopper to search the defined chan S e the displayed images resulting in displaying through 
searchable directory in that the shopper types in one or more & c monitor of the computer 20 a changing video represen- 
words to identify the product to be located (e.g., "green 55 tation of the configuration of displayed products correspond- 
beans"). The system then matches this to the stored directory m S to a movin g me shopper would see in moving from 
listing and retrieves and displays back to the shopper th e first position to the second position in the actual shop- 
through the screen of the computer 20 where the product is P m & fr^ty- 

located in the store 2. This can be by a text response (e.g., The method of the present invention also includes dis- 

"aisle 2") or by displaying the video image of that portion of 60 tioctively displaying predetermined ones of displayed prod- 

the store as would be seen if the shopper moved among the nets to indicate to the shopper that the distinctively dis- 

virtual aisles. The shopper can select these found items, played products are offered on a special basis or need to be 

make one or more lists, and order if desired, all as indicated reordered or are on the current order list, for example, 

in the flow chart of FIG. 10A. The shopper then can print the The method also includes indicating to the computer the 

list(s), "go shopping," or return to the home page 24, also as 65 shopper's selection of a specific product. This can be by, for 

shown in FIG. 10A. Other specific implementations can, of example, clicking on the displayed product, or clicking on 

course, be used. . the displayed product and dragging it to a video represen- 
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tation of a shopping cart, or answering a selection query in 
a display window on the screen. Selection can also be by a 
light pen, screen touch, keyboard actuation or other means. 
When a product is selected, it is automatically added to the 
selected product list. 5 

The invention also includes maintaining in the computer 
a historical list of products selected by the shopper. 

From the product selections, a list is created in the 
computer 18 or 20, which list contains the products to be 
purchased by the shopper at the actual selected shopping 
facility. The list can be organized in a sequential order in 
which the selected products are to be picked up at the actual 
selected shopping facility or at another location such as a 
warehouse where electronically transmitted orders are filled. 
Other list organizations can be used. 15 

When the list has been completed, it can be printed for the 
shopper to use when the shopper goes to the actual selected 
shopping facility, for example. 

As mentioned above, the hardware and software for 2 o 
implementing the system and method described above are in 
general conventional and known in the art. To the extent any 
modifications or additional hardware or software is needed 
in specifically implementing the invention, such modifica- 
tions and additions are within the skill in the art given the 2 s 
explanation of the invention set forth above. 

Thus, the present invention is well adapted to carry out the 
objects and attain the ends and advantages mentioned above 
as well as those inherent therein. While preferred embodi- 
ments of the invention have been described for the purpose 30 
of this disclosure, changes in the construction and arrange- 
ment of parts and the performance of steps can be made by 
those skilled in the art, which changes are encompassed 
within the spirit of this invention as defined by the appended 
claims. 35 

What is claimed is: 

1. A food vending method, comprising: 
maintaining an actual operating food vendor at a physical 
location, the food vendor having a customer service 



area; 

converting at the customer service area actual images of 
the customer service area and food served therein into 
encoded digital signals; 



40 



storing the encoded digital signals in a computer storage 
medium; and 

providing for remote computer communication link 
access to the stored encoded digital signals such that 
transmissions over the communication link responsive 
to the stored encoded digital signals display on a 
monitor of a computer at a location remote from the 
physical location of the food vendor a continuous 
motion visual representation of movement and food 
within the customer service area enabling a person 
viewing the monitor to have a visual experience of 
walking in the customer service area of the actual 
operating food vendor, sitting down at a table, and 
opening a menu as if the person were in the physical 
location of the actual operating food vendor. 

2. A method as defined in claim 1, wherein: 
converting actual images of the customer service area 

includes converting an actual image of a menu used in 
the customer service area; and 
the continuous motion visual representation includes the 
actual image of the menu. 

3. A method as defined in claim 2, wherein the food 
represented in the continuous motion visual representation 
appears in response to selecting at the computer the respec- 
tive food item from the actual image of the menu in the 
continuous motion visual representation. 

4. A method as defined in claim 2, further comprising 
transmitting an order for food from the computer to the food 
vendor over the communication link. 

5. A method as defined in claim 3, further comprising 
transmitting an order for food from the computer to the food 
vendor over the communication link. 

6. A method as defined in claim 3, wherein the selected 
food representation appears in continuous motion showing 
the selected food being served at the table. 

7. A method as defined in claim 1, wherein the continuous 
motion visual representation further enables the person 
viewing the monitor to have a further visual experience of 
seeing a food item being served to the displayed table after 
the person has selected the food item from the opened menu. 
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